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ABSTRACT 

The future of instructional research, at least in the 
present economic climate, is indistinct. This report considers the 
.option of combining within a single study the needs of policy makers 
and the commitment to academic research*. The papers in this report, 
through illustration of research conducted within a policy framework, 
identify problems and/pr benefits of the forced marriage of knowledge 
production and decision-directed research. Methodologies for 
optimizing the match are explored. In each case example, the research 
focuses on classroom behaviors and related instructional activities. 
Outcomes of interest include cognitive performance and affective 
responses from students and ^teachers . The report considers future 
directions of research, not only as suggested by. the specific 
findings of theoretically derived inquiry, but also as such options 
may be influenced by the reality of political, administrative, and ^ 
economic constraints. Edys S. Quellmalz identifies probletas and 
limitations of current designs for serving instructional Research * 
needs, and Suggests some alternative research strategies, foan L. 
Herman presents methodologies for combining research and policy 
needs, and suggests the advantages inherent in their merger. Finally, 
Don Dor r-Bremme .highlights the advantages and problems involved in 
embedding a piece of instructional research in .a larger policy study. 
-(Au€hor/PN) • * , 1 
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INTRODUCTION 

'The future of instructional research, at least in the present 

economic climate, is indistinct. The. trends suggest a continuing » 
reduction of support for basic research jtnd a concomitant increase in 
competition for scarce resources. At the same time, evaluation or , 
other policy directed' studies may continue at their present level, if 

•for no other reason than to provide rationales for .budget reduction. 
This report^ considers the option of combining within a single study 
the needs of\ol icy* makers and the commitment to academic research. 
The decisions to be made involve real vs.. laboratory settings, 
experimenter controlled vs. naturalistic designs, lean vs. thick data 
collection, and' political' reality vs. scholarly quality. The papers 
in this report, through illustration of research conducted within a 
policy framework, -will identify, problems and/Or benefits of the forced 
marriage of knowledge production and" depi si on-directed research. 

' Methodologies for optimizing the match will also be explored. In each 

> 

case example, the research focused on clas'sroom behaviors and related 
instructional activities. Outcomes of .interest included, cognitive 
performance and affective responses' from students and teachers., 

■ The report considers future directions of research, not only as 
suggested by the specific fidings of theoretically derived^ inquiry,- 
but also as such. options may be influenced by the reality of 
political, administrative, and economic constraints. How can we serve 
self-interest, research, and policy interests? .For example, the - 
values of academic freedom come in direct conflict with centralized, 
e.g., policy, mandates. . : 

j 
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In the report, Quellmalz Identifies problems and limitations of 
current designs for serving instructional research needs, and suggests 
some alternative research strategies. Herman presents methodologies 
for combining research an$ policy needs, and suggests the advantages 
inherent in their merger. Finally, Dorr-Bremme highlights the 
advantages and problems involved in. embedding a pierce of instructional 
research in a larger policy study. 




ISSUES IN DESIGNING INSTRUCTIONAL RESEARCH; EXAMPLES 
' . FROM RESEARCH ON WRITING COMPETENCE 1 

* • kdys S. Quellmalz • , * 

Instructional research ranges from broadly conceived national studies 
of schooling's effect, on basic skills achievement to individual researcher's- 
studies of specific variables promoting particular skills. Most of these 
studies tend to focus on features- of the school , ^lassroom, teacher, and 
curriculum to -identify policies^ actions facilitating learning. There, 
are widely divergent perceptions, however, of the type and specificity of - 
independent and dependent 'variables appropriate in large-scale (top down) 
and small-scale (bottom-up) studies of ' instruction conducted in the school 

i 

context.' 

Much large-scale research is driven by evaluation methodology, while 1 

t 

smaller-scale studies use paradigms from instructional technology and 
cognitive psychoT&gy. This paper -describes two main categories of problems' 
that seem to pervade school-based studies of instruction: The first set 
of problems relates to the design of outcome measures in term's of (1) the ; 
lack of sensitivity 'of many dependent measures used to document instruc- 
tional effects, (2) the failure to collect corroborating measures of 
effect, and (3) the failure to match the content and processing require- 
ments of alternative measures witH each othef. The second category of 
problems is in^ the design of context and process descriptions ; including 
> (1) the failure to describe contextual dimensions of the school and 
curricular systems that set the conditions within which instruction occurs, 



(2) the failure £o freely, explore instruction as an interactive process; 
to relate instructional variables to logical or research bases to explain 
achievement results, and (3) the failure to .compare the context and pro- 
cessing requirements of classroom tasks with test tasks. 

The purpose of this paper is to describe features of instruction — 
' its contexti processes, and outcomes — whose relevance and utility for 
instructional improvement seem %o have the strongest empirical support- 
The paper argues that researchers and ^valuators studying instruction in 
schools should sharpen the focus of measures and bettter trace the inter- 

- relationsh:ips.,within and between independent and dependent variables, 

* . ' / 
. Problems in the Design of Outcome Measures 

Lack- of sensitivity . A prevalent problem in school -based instruc- 
tional research is that test tasks are often insensitive to the logical 
and psycho! ogfc'al aspects of tasks presented in instructional interventions,- 

, There is a gap between notions of the appropriate level of detail for 
describing and constructing .dependent measures in laboratory-based 
instructional research designe4 by psychologists, on the one-hand, and in 

' school -based instructional research conducted by evaluator^ and psycho- 
metricians , on the other hand. For e^amp^ federal evaluations 
?nd many state and* district evaluation studies still r61y primarily or ex- 
clusively on norm- referenced tests to detect instructional "effects. The 

' many criticisms of norm- referenced tests^ for reflecting achievement of spe- , 
cific instructional goals have^een described elsewhere. (Glaser, 19§3'; Popham 
1978;< Millman.,' 19>.4; Hambleton .et al\ , 1978), The recognition of the nee<± 
for a much closer match .between,, testing and instruction first , stimulated 
the' call .for crita^on-referehced -testing (Glaser, 1963.), Furthermore, 
a growing body of learning research shows that student^ performance varies 

\ ■• ■ * 
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significantly when task demands change. Classes of task or problem types 
require students to access different bodies of stored information and to. 
activate different procedures, routines, or solution strategies. 

For- example, in math, the work of John Seely Brown and his associates 
demonstrates that different sets of subtraction problems elicit different ' 
solution schema (Bro,wn et al., 1978). Thus descriptions of achievement -at 

* 

the molar level of "math achievement" or even of "computational skills" can- 
not sufficiently describe performance on homogeneous sets of skills, nor ■ 
signal skill areas requiring attention at the program, classroom, or individ 
ual level. Similarly, reading research indicates that' reading comprehension 
is' not an undifferentiated construct; rather the type or discourse mode of 
reading material, j(uch as narration and exposition, requires different ' 
schema for comprehension (Brown et al . 1978; Meyer," 1975). This research, 
implies that, if ( te^s are to be sensitive to different types of, reading . 
skills, they must be designed to provide subscore profiles on skills or ■ 
inferencing required by different types of reading passages. They cannot 
merely report. generalized scores for decoding and literal and inferential 
comprehension. Yet federal-level evaluations such as Follow-through and 
Cities in Schools (Murray et al.,-1981) report global "reading achievement" 

scores. s 

Nowhere is -the insensitivity of dependent measures, more dramatically 
illustrated than in the recent surge of studies of-- writing. Like reading 
achievement, writing achievement must be decomposed into the level of 
skill demonstrated in relation to different types of writing 'tasks. The 
* various controlling purposes of discourse modes or genre require students 
to use different kinds of topical information and -different presentation 
strategies according to organizational schemes and 'development methods 
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conventionally expected in these various genres. 

Large-scale evaluations of writing competence have too frequently 
used multiple choice jtems"to measure, writing achievement. Psychologists ' 
would deny the contruct validity of recognition tasks as anything other 
than enroute indicators pf production capabilities. Research on the 
comparability of information derived from indirect (multiple cho'ice) "and 
direct (writing' sample) tasks has primarily been conducted by psycho- 
metricians more schooled in metrics than .psychology. Studies condupted 
within the psychometric framework have reported high correlations between 
total multiple choice test scores and holistic essay scores and cite these 
correlations as support for substituting multiple choice' tests for essays.- 
Recent research within a competency .testing framework has investigated 
the comparability of information from these two measurement response forms. 
'They have found lower total score correlations and, more importantly, 
-much lower correlations between direct and indirect scores for subskills 
such as coherence, support, and mechanics (Moss, Cole & Khampalikit, 1982; 
Quellmalz & Cape 11, 1982; Quellmalz & Baker, 1981). / 

Studies of the effects of instructional inter>tff£ions on writing 
achievement also demonstrate that holistic scores do not adequately de- 
scribe how the varying- skill levels. in component features of the product 
contribute to-the global quality score. For example, studies guiding * 
students in writing strategies may. find no significant differences in 
pfe- and post-intervention judgments (e.g., Pearl, 1979), yet re- ^ 
searchers discussing these inconclusive results cite observational 
information suggesting that student writing really did improve. At a 
conference of grantees of federally funded writing projects discussing 
their research progress, a dominant concern was the failure of holisitc 
' f 
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essay ratings to. capture improvement or, at least, changes in student ' 
"writing due to instructional treatments. The remedy, of course, is to 
design scoring schemes that include criteria, subskill ratings, and even 
detailed secondary discourse analyses detailing the features of interest. 

Some writing researchers are doing this.- Odell (1978) instructed 
studehts in Pike's (Young, Becher, & Pike, 1970) pre-writing discovery 
approach for selecting and organizing essay content. While/judged essay 
quality as a whole was not affected, textual analyses shd^d that students 
use of rhetorical devices such as temporal sequence and classification did 
{ increase. Similarly, Bracewell, Bereiter, and -Scardamalia (1980) taught 
^tudents rhetorical strategies for persuasion and found their use signifi- 
cantly increased, although overall ratings of essay quality 'did- not. If 
holistic essay quality scores were the only ones reported, it might be con 
eluded that the instructional interventions had no effect and should be 
.dropped. But when more detailed analyses document tha,t what was taught 

4 < 

was used.in students 1 writing, t«he implication may be that detection of 
overall quality effects requires" . , 

• more time and practice » 

. additional and different instruction on the subskill to help 
students use it more effectively 

• instruction to help students integrate, the strategy with other 
^ writing skills. 

An analogy is seen in the case of a tennis ffistructor working with 
a student on his/her backhand. At the end of a series of lessons, two 
dependent measures seem appropriate: (1) is the backhand stroke and 
resulting ball- placement better? (2) does the student-win more games? 
If only the "games won" measure is used, it might be concluded that: 
(1) the backhand instruction had no effect and should not be used again, 



"(2) the student needs more practice time, (3) while concentrating on his 
backhand, the student's forehand went to pot, contributing to the "no 
increase in games won" score. This last phonomenon, all too familiar , to 
athletes > implies the need for more practice and, most likely, for 
instruction on integrating use of the two strokes. 

Therefore, -many instructional studies would profit from more careful, 
detailed designs of dependent measures that document performance on,.sub- 
skills taught, as well as overall performance. -Policy decisions at federal, 
state, district, and classroom levels which draw exclusively on the over- 
alT measure might conclude that the program or treatment had no effect at 
all.* The treatment, however, might have been effective., but the measure 
was too gross to detect it. 

Failure to collect corroborating outcome measures' . A second problem 
•in'the design of outcome. measures is the failure to collect information 
about student performance on other facets of the skjTl. Too often, 
evaluation and instructional research studies report global performance 
on a single measure such as a math achievement or a reading achievement 
test. Again, writing assessments that collect only one sample dramatically 
illustrate the illogical and methodologically unsound nature of the "one- 
shot^, performance index. Numerous' studies of writing assessment demonstrate 
fluctuations in individual performance on different writing tasks (Crowhurst 
& Piche, 1979, Quellmalz & Capell, 1982). Certainly, we^feel increasingly 
confident about students' competence, when it is demonstrated repeatedly. 
Unfortunately, most commercially available tests present only one or two 
items per skill: While multiple performance indicators on Other formal 
assessment devices are helpful, research* suggests that progress on in- * 
class work samples may provide better corroborating data. Studies of 
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test anxiety and contextual influence on performance, especially on writing • 
performance, support the utility and validity of collecting classroom per- 
.formance information, since the classroom is the more realistic and normal • 
context for the student.. 

Failure to" relate the context and processing requirements of alternative 
measures to each other . When con^0tfo>aiLng-data are collected, it seems 
reasonable that the data should be from performance on tasks similar in their 
processing*requirements. Yet studies often fail to check or describe whether 
i, types of tasks on two formal tests or on classroom problems match. Whether 
direct records of classroom assignments ^md test performance are collected 
or teachers' indirect ratings of achievement progress are gathered, the 
comparability of tasks must be rigorously described. 

Problems in the,Design of descriptions of Instructional Contents and , 
Processes S 

Alternative research paradigm's focus on the learning environment's 

features that differ substantially in specificity and in proximity to 

the learning event. The search for effective instruction in the complex 
r 

formal school setting has, fortunately, grown from studies of teacher 
personality and vaguely defined teaching methods and now includes political 
and administrative contextual influences of the extended school system.' 
Also, recent research' is attempting to document the classroom's physical," 
social, and managerial context to explain factors influencing the inter- . 

< 

active information processing of teachers and students. Few studies of 
instruction, large-scale or classroom level, trace the links between 
the; conditions under which instruction occurs, features of the instructional 
process, and learning outcomes. While- experimental researchers conducting 
laboratory studies are trained to describe the' conditions in which treat- 
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ments occur, researchers studying .the .complex classroom environment may ,fail' 
to describe extra-classroom conditions' that constrain instruction. , Similarly, 
large-scale evaluations may fail to describe alternative conditions of im- 
plemetitation^that relate to program effects. Federally funded evaluation 
repo/ts sometimes describe program implementation (process and context) in 
sep/rate volumes. Policymakers thus find it difficult to trace cause/effect 
relationships between instructional implementation patterns and achievement 
/ata (e.g., Murray et al., 1981). 

The context and process variables affecting learning outcomes are 
broad" in scope and large in number. Figure. 1 suggests -categories of 
contextual and process variables with a research base supporting their s 
influence on achievement. f The contextual variables includ? constraints ^ • 
imposed by the existing organizational and curricular systems as well as 
by teachers' and students' entering perceptions and^ abilities. Variables 
involved in the course of the instructional process include the inters 
actions of task features, teacher behaviors., and student behaviors in 
the hypothesized internal learning processes of students." 

The system context: the school . Studies of effective schooling 
have identified policies enacted at higher levels of the educational and 
political system that profoundly affect the ultimate nature of classroom 
instruction. For example, legislative mandates affect the composition 
and stability of the school population. Funds allocated for the support 
of general education and spec i£U> programs influence the range of avail- 
able resources, usually personnel and materials. Resource allocation 
pol'icy decisions seriously affect potential instructional quality - 
(Harnishfeger & Wiley, 1976).- Perhaps the most influential legislation 
affecting school instruction has been minimum competency testing re- j 



* Figure 1: 

Issues In Designing Instructional Research; Examples from Res earch on Writing Competence 



FRAMEWORK OF VARIABLES fOR STUDYING INSTRUCTION 



System* 
Context 

Federal funding 
regulations 

Laws on school 
student compo- 
sition- 

Laws on minimum 
competency 

01 strict policy 

School adminis- 
trative policy, 
emphasis, and 
style 



CONTEXT 

CurHcular 
Context 

Mandated 
goals 

objectives 
syllabus 

Available ( 
test ' \ 
materials 
resources 

Available 

] curriculum* 
consultants 
staff de- < 
velopment 



VARIABLES 
Teacher 

Characteristic s 

Expedience . 
y\ teaching 
In the subject 

Orientation 
teacher 1 s role, 
preferred 
teaching 
methods 

Concepts of 
subject matter 

Information base 
about subject 
matter 

\ Judgments of^~ 
* ' student ability 

Expectations for 
^ student progress 



Student 

Characteristics 

Ability level 

Achievement 
level docu- 
mented 

reading 

writing 

math 

Language 
development 
(other dominant 
language) 

Values and 
expectations 
about subject 
matter 

Cultural 
background 



Task Features 



Social context 

Functional purpose 

Relationship to 
S's world 
Knowledge 

Structural 
features 
kind of task 
and Inter* 
relationship 
of Information 
presented or 
, required 

Required processes 
Information 
component strat- 
egies 
patterns to Inte- 
grate above 



— -Process variables 
interactive instruction 



Teaching Behaviors j 

Goal setting 
Describe outcomes* 
content, form 

Orient to relevant 
features 

Presentation/Explanation . 
Present or elicit 
relevant content, 
strategies, rules 

Feedback* 

Practice . 
EUdt response [ 
AsIc questions < 

Feedback 
On appropriateness 
of details 

codes , 

procedures 

strategies 

via 

praise 

confirm 

correct 

Induce - 

tell 

give explanation 
rule 
example 

Patterns of all above 
t points of application 
'orchestrated and 
integrated 



Student Behaviors 

Physical orientation 
eyes, body 

Ask questions 

Rehearse > 
segment 

label /categorize/ 

Elaborate/transform 

Relate to other 
knowledge 

Use Imagery " 

Answer questions 

Verbalize rules, 
strategies 

Solve problem 
plan 
write 
revise 
edit 

Patterns of all above 
orchestrate and 
Integrate 



Learning 
Process 

Attention 



Encode 



Retrieve 
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quirements and the accompanying scrutiny of the quality of instructional 
opportunities preparing students to pass those exams. Testing require- 
ments influence curriculum emphasis and student achievement (Yeh, .1978; 
Schwille, Porter, & "Garet, 1979; Floden, Porter, Schmidt, & Freeman, 1980). 
Also, administrative policy within the educational system, such as curric-: 
ulum guidelines and state adopted texts, constrain classr«om optjons. 
Edmonds (1979) cites several studies (e.g., Weber, 1971; New Yor*, 1974; 
Brookover & Lezotte, 1977) showing fcftat an active and supportive school 
administration leads to higher achievement levels i\>nner city schools. 
These studies point to the need to describe the systemic and curricular 
•conditions limiting instructional options in the classroom. 

Within the classroom itself, teacher effectiveness studies conducted 
through observations and naturalistic inquiry identify ^administrative 
policies that contrain instructional options. Number of p.upils,*ssigned 
to each class and range of pupil ability in a syigle classroom dertainly ■ 
bound- teachers ' planning (Dahllof, 1971}. i 

As Barr and Dreeben (1977) have noted, studies of classroom effects 
must refer to the broad social context in which the classroom functions.. 
Their reviews of instruction in classrooms and Doyle's (1977) critique 
of paradigms for research on. teacher effectiveness underscore the, need 
for expanding the breadth and depth of variables considered when examining 
classroom ecology. They contend, and' I agree, that most current research 
.paradigms fail to consider^ the full range and the functional interdepen- 
dence of contextual variables in instruction. 

The system context: the curriculum . .Classroom research across sub- 
ject matter suggests that the basic unit of instruction is the assignment 
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or task. ThHask is described as a goal -directed set oA activities 
presenting*. students with .content of certain characteristics and required 
procedures for completion (Doyle, 1979; Mehan, 1974; Van Nostrand et al . , ~f 
1980). For teachers,, tasks involve -content, materials, and activities' 
(e.g., Morine-Dershimer, 1979; Shulman, 1980; Schutz, 1980). Research . 
suggests. that materials availability' strongly influences the types of 
problems, practice,, guidance, and feedback that students receive. .Ob- 
servations of classroom instruction for low achieving, low SES students 
in the elementary "grades revealed that students spent as much as 70% of 
their time working alone with materials. Although studies' at the second- 
ary leveV indicate that activities* are less materials-driven (Sirotnik, 
1981; Applebee, 1981; Van Nostrand et al . , 1980), it may be that appro- , 
priate materials are less abundant, or unavailable. In any„case, the in- 
structional quality 6f commercially available materials has been criti- 
• cized severly •(Quellmalz et al.. , 1977; Van Nostrand et >T. , 1-980). 

In any-Jtudy of effective instruction, then, one category of central 
questions shjbuld address the availability and quality of curriculum materials. 

The interactive instructional process . When the teacher effectiveness 
literature is viewed from the perspective of theories of learning and in- 
struction, mapy findings are rendered irrelevant or useless to the design 
of instructional. research. Characteristics such as "businesslike" are too 
fafremoved from the refinements of student information processing. * Medley V 
(1979) extrapolated effectiveness constructs certainly indicate that many 
descriptive studies* are far removed from the learning act. For example, 
maintenance of a learning environment includes both "orderly" and "support- 
ive" behavior. "Time on. task" was reported effective in large, group set- 
. tings only. Methods of instruction generally thought to be important 
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were found to be ineffective with disadvantaged learners. Among these 
methods were high level questions, students asking many questions, pro- . 
viding moreteedback, and increased teacher amplification. ^ 

It is- clear that the analysis of teaching and learning must provide 
much more detailed descriptions of the' conditions under which 'such findings 

-A 

prevail: Peterson (1979) notes the highly contingent interdependences 
of instructional yariables in her critique of Rosenshine's review of the 
effectiveness of the direct instructional model (Rosenshi'ne, 1979). Rosen- 
shine" identified major components of this model as: (1) clear- goals, 
(2 J sufficient and continuous goals, (3) content coverage, (4) monifcpHng 
of performance, (5) low cognitive level questions, (6) imnediate academically 
oriented feedback. Studies reviewed by Medley and Rosensnine focused on dis- 
advantaged elementary age children. Thus, one might guess that low level 
questions were better predictors of performance because students were 
just learning skills and. because low level items were on the test. 

Teacher effectiveness studies concentrating on "time on task" have 
primarily been large-scale Uee Cooley & Leinhardt, 1980; Fisher et all, 
1978). Findings about academic learning time were not startling? what 
was surprising was how little classroom' time was provided for learning^ 
tasks Clearly classroom time management is prerequfsite to effective 
participation in instruction. In the Instructional Dimensions Study, 
•within the instructional event, the techniques identified as related to 
/ the quality of instruction were (1) focusing attention on the task, (2) ref- 
erring to previously used material, (3) referring to earlier performance, 
and (4) effective classroom management. In the Beginning Teacher Evaluation 
Study, teaching methods associated' with achievement and academic learning 
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time were {1} provision of tasks permitting a high success rate, (2) more 
presentation of information, (3) more monitoring of work, and (4) more ' 
feedback about academic performance. These findings coincide with those 
reported by Stall ings (1980), who describes interactive on-task instruction 
time as characteristic (of effective -teachers. Effective instructional 
patterns were (1) more support and (2) positive corrective feedback*. The 
nurturing environment is particularly iijiportant for secondary students. . 
with a history of failure. The need for positive, informative modes of 
feedback has also been reported. For example, Webb (1980) found that 
students working on cooperative tasks in groups participated more actively 
and achieved more when the group gave and received more explanation about 
how to solve problems. • \ ' 

While these teaching behaviors apparently can and do occur, some\ 
researchers using naturalistic inquiry methods report that teachers may 
leave performance expectations unsignaled, i.e., no clear goal setting, 
(Mehan, 1974') and provide inconsistant feedback. The question to be 
asked is how effective teachers plan and contruct instructional events 
to result in effective interactions. Borko et al . (1979) suggests that ac- 
tual teacher planning is at odds with the* idealized paradigm. As men- 
tioned previously, teachers plan in terms of content, materials, "and ac- 
tivities. The resultant instructional task or assignment becomesthe^ 
basic unit of planning and action in the classroom (Doyle, 1977; Clark & 
Yinge'r, 1979). A major line of ethnomethodological and sociolinguistic 
inquiry focused on descriptions of teachers' decision-making for planning 
and aid during the interactional teaching-learning phase. One example of 
the detailed level of this research is reported by Dorr-Bremme. In a 
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fine grained sociolinguistic analysis of daily classroom events, he showed ( . 
that teachers adjusted their style in subtle, but extremely important ways 
in response to how students spoke, and 'acted .(Dorr-Bremme, 1982). 

I 

In sum, the teacher effectiveness literature suggests the need to 

r 

include several variables in the design of instructional research: (1) 
broad school. and district level contextual factors affecting resources, 
required content, emphasis, and'materials availability, (2) classroom 
context factors, including student perceptions and teacher-student inter- 
actions/and (3) teacher factors including decision-making-, planning, and 

class "management. ' • 

: Linking context and process with outcome . Clearly there are a large 
'number .of contextual and process Variables that potentially influence' 

^learning. Large-scale studies of instruction, such as the Instructional • 

t 

Dimensions and^ Beginning; Teacher Evaluation studies (Coo ley & Leinhardt, 
1980; Fi,sher\et al l, 1978), have 'attempted to collect a range of information 
on context and process. *The analytical problem'comes in first assuring 
that tasks or 'items in outcome, measures are similar to those occuring in 
instruction. A second* major- methodological problem involves identifying the 
configurations of contextual 'and process .variables that affect achievement 
data,* Studies o( writing, for exampJe, are beginning to r*evea"l that stu- 
dents really aren't writing much at al| (Graves, 1978; Applebee, 1981; 
Pitts, 1978) and surveys of writing instruction .are indicating that stu- 
dents who do write receive little guidance or feedback: ; In addition, the 
kind of writing students produce in* class may differ both in form and dis- 
course mode from that tested (Quellmalz, Baker, & Enri.ght, 1980). Thus^' 
the policy implications of test evidence that students aren't writin^may, 
differ markedly depending on the context and process data. 



Summary and Conclusions , - 

I have suggested .that the design* of school-based studies of instruc- 
tion have strayed from the rigorous methodalogy required, for' social and 
scientific research. In particular- I tiaVe argued that the methodologies 
of large- and small-scale studies could strengthen the des.igns of^ their 
outcome., context, and process measures and the relationships among theun » 
Psychological research methodology requires rigorous' and replicable 
descriptions of dependent (,outcome) and independent (process) variables 
and the conations (context)Jn which they are studied. Some may argue 
that the ngor of laboratory methodology cannot or should not pertain to* 
research in the complex school environment, I disagree. School-based 
Instructional research can construct* and x use outcome measures logically 
and psychologically sensitive to instruction and also collect corroborating 
performance tfat£ on comparable tasks. While the- range and number of con- 
text and process variables may seem intimidating„/a research study may 
gather and report the situational specifics (context) and ^instructional 
processes relatively inexpensively through researchers' informal obser- 
vations, or more expensively through formal interviews ^ questionnaires, 
and structured, observations' or enthnographiers. Key criteria for context 
and process variables are that they (1) relate logically and .psychological-/ 
'ly to student learning, <2) can be clearly and replieably described, and 
C.3) are amenable to instructional and administrative action. The pivotal 
: design issue will be to think harder, plan carefully, and trace sensible 
relationships withrn the data gathered in. an attempt to provide explanations 
of achievement. More attention to the design of instructional research 
would . avoid the' expensive, useless data gathered by researchers who may be 
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untrained or unthinking. • • 

\ Sensitive outcome measures can report' students 1 performance on a 
reasonable number of items or tasks (not one or two) measuring subskills 
as well as total scores. Criterion-referenced testing programs are at- 
tempting, this now: Data from a sensitive test can always be aggregated , 
or disaggregated at a 'level appropriate to 'the policy decision (individual, 
"class, -school, district, state, or nation). Data from an insensitive test, 
can never be disaggregated or decomposed. For example., policy makers are , 
better served by 'data indicating the type- of sut*ski]ls on which students 
have difficulty, rather than a statement that they "can't read." Finan- 
cial resourced can then be focused on curricular and personnel selections' 
relevant to areas of performance weakness. 

Data collected through teacher records of performance on^glass assign- 
ments and tests can corroborate -test information. By improving the designs 
of instructional research, projects' limited research funds should yield 
.more valid, useful information for improving instruction. . • 
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MERGING POLICY AND RESEARCH INTERESTS:, A CASE FOR MUTUAL NEEDS 

Joan L. Herman 

i 

Introducti on 

Social science research dollars ,are dwindling and the outlook for 
sponsored research grows dimmer on a dally basis. But while the, 
picture 1s bl.eak, their may be a faint light on the horizon** The 
continuing need for and commitment to evaluation research may brighten . 
some of our futures. 

Funds allocated for evaluation and policy studies have Increased 
dramatically 1n the last decade, and while such escalation 1s unlikely 
to continue, available funds may hold their own— a marked contrast to 
the outlook for other social science research. The evaluation funds - 
currently tied to bloc grants, for Instance, are hartjly Insignificant, 
and the'emphasls on local rather than federal program evaluation 
Increases their appeal. Can educational research find a home, health, 
and happiness with these^avallable dollars? Perhaps. Certainly some 
compromises will have to be made, but evaluation studies can serve 
some mutual needs of Instructional researchers and of policy makers, M 
and the .merger can benefit both, parties. . 

Evaluations, after all, can be conceived as hypo thesis- testing ^ 
ventures. That 1s v , consider the proposition that many special 
programs, especially school reform efforts*, are social experiments _ 
which, among other tfcHtgs, attempt to, translate research Ideas Into 
practice to achieve particular outcomes. For example, California's 
School Improvement program and Its predecessor,. Early Childhood 
Education, as well as many federal educational programs, are based on 
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a number of premises about what factors contribute "to and foster 
school effectiveness and student achievement; e.g., the efficacy of 
parent Involvement, systematic planning and evaluation, lower 
adult-student ratios, Individualized Instruction, etc. More 
straightforward examples are the FollbwthpoTigh programs, whlfih are r 
based' on fairly specific models of how Instruction ought to occur. « 

Given the perspective that educational programs embody, or at 
least Imply, particular treatments, then the task of evaluation 1s to 
test the' hypothesis t»at the specified treatment 1s, 1n fact, 
associated with the desired outcomes. The applications of research 
methodologies and notions of operational 1z1ng and measuring the 
Independent variables as well as the dependent variables ^are obvious 
here, as are the potential relationships between legitimate evaluation 
questions and research questions. While evaluation conducted 1n a 
real -world setting may be sloppier than work conducted 1n more * 
controlled research environments, the principles are largely the same. 

Obviously, 1f you want to know whether a treatment works, or 
■ whether an 1-ndependent variable has particular effects, sound" research 
design suggests that you first define the treatment,' and then make 
sure that 1t 1n fact occurs. You can't 'evaluate the effects of an' 
empty set nor draw Inferences about the results of an absent 
treatment. We know thfs— It's obvious— but evaluations often miss 
this essential point. Too many evaluations try to answe'r the question, 
iDoes the program work?" without first asking "Was there a program?" 
This practice may occur because of client unsoph1st1cat1on 1n research 
and evaluation design and lack of Interest 1n program processes. 
Program managers and operators ask simple questions and want s1mpt& 
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answers. But they cart, be convinced of the need for more. For 

Instance, 1n our early experiences 1n evaluating California's Early 

Childhood Education Program,, the funders Initially were Interested 

only 1n outcome data. However, we at CSE held flmi on the needs for 

process data arid were permitted to proceed as we desired. 

* • 

So, evaluation 1s, and ought to' be, concerned with school process 
as well as outcome, and for programs aimed at student achievement, ' 
It's not hard to bring evaluation studies Into the classroom. That 
1s, 1f you agree that student achievement 1s principally a function of 
what teacher and students do 1n the classroom, 1t 1s easy to build 
the case for why evalua&on ought to look at Instructional practice. . 

Where does this take us? In place of the single basic question 
"Is the program effective?" sound evaluation will ask: 
1 1. "What 1s the treatment Implied by the program? 0 

2. To what extent 1s the treatment Implemented? 

3.. What are the outcomes of the program? 
If the answer to question two 1s positive and there 1s a demonstrable 
treatment, then the outcome data g1Ve a valid answer to the Initial 
basic question— I.e., does the program work? But aligning answers to 
questions two and three provides food for Instructional research and 
asks "what are the effects of the treatment and to what extent do ^he 
Independent variables affect the dependent variables of Interest? J 1 
Such process-product research has been with us for Some time, with a 
somewhat checkered history, but prior specification and newer causal 
modeling techniques can Increase Its power. Let me provide an example 
of . how evaluation provides opportunities for Instructional research. 
The Example 1s Imperfect, but does demonstrate how educational 
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evaluation can contribute to our knowledge base. 
An Example: A Study of Individualized Instruction < 
The Center for the Study of Evaluation conducted a study of 
California's Early Childhood Education (ECE) Program (Baker, 1976). 
This* program, according to the then current section 6445 of the* 
California Education Code, provided that early childhood education be 
designed, among other things,* to assure: a 

(a) a comprehensfve restructuring of primary education 1n 
California, kindergarten through third grade, to more fully 
meet the unique needs, talents, Interests, and abilities of 
each child. 

(b) the cooperation and participation of parents 1n the educa- 
tional program to the end that the total community 1s 
Involved 1n- the development of the program. 

(c) that pupils participating will develop an Increased 
competency 1n thfe skills necessary to the successful 
achievement 1n later schoql subjects such as reading, 
language, and mathematics. 

Thus, one could reasonably Infer that, the ECE program was Intended to 
foster student achievement through, among other things, more^ Individu- 
alized instruction and' community involvement, I.e., prograft processes 
were means to an end. Alternatively one might take a more coordinate* 
view that higher student achievement and more individualized 1n£truc- 
tlon for students were equally valued program outcomes. In any case, 
1t was clear that Individualized Instruction was an Important 
component of ECE and, consequently, CSE's study collected a range of 
questionnaire and observation data about how teachers Implement 
individualized programs. In addition, because ECE claimed an interest 
' in the "whole child," criterion-referenced tests of reading and 
mathematics as well as measures of students' attitudes werfc also 
collected. The data set allowed us to Took at how Individualized 
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Instruction operates 1n classroom practice, and to examine Its affect 
on student outcomes. , 1 

The secondary analyses posited a model to explain expected Inter- 
relationships between. attributes 'of Individualized Instruction and 
their direct and Indirect effects on second grade students' achieve- 
ment and attitudes. The underlying assumption of the hypothesized ^ 
modeT was that classrooms which were more Individualized 1n terms of 
Instructional decisionmaking, activities, and teacher-student Inter- 
actions would provide more appropriate Instruction for students and- 
thus result 1n Improved student achievement and attitudes. It was 

m 

also assumed that 1f an Individualized program was Implemented ■ 
systematically, the degree. of Individualization 1n, decisionmaking, 
activities, and Interaction with the teacher would be Interrelated. 

* 

That 1s, Individualized decisionmaking would lead to different 
prescriptions and ,d1 f ferent kinds of Instructional activities for 
different students," based on assessment of need. Further, having more 
activities going on 1n the classroom should allow the teacher to 
Interact on a more Individualized basis with students working on any 
single activity. Aides and volunteers were conceived as serving a 
support function in the classroom, I.e., their presence allowed 
teachers to manage the Individualization effort. Socio-economic 
-status was also Included as a control 1n the model as well as to 
examine Its effects'. Path/analysis was used to test the direct and 
indirect effects predicted by the model. 
ThfrPata Set 

The data used for the analyses were. a subset of those used for 
the main ECE evaluation. A stratified random sample of 256 schools 



were selected for participation 1n the main study to represent three 
.levels of ECE status {Q, 2, and 3 years) and four levels of compen- 
satory education funding (receipt and non-receipt of federal and/or 
state level funding). From within these 256 schools.* 72 were selected 
for more Intensive study. Two second-grade and two third-grade class- 
rooms within the 72 schools were randomly chosen for data collection. 
The study of Individualized Instruction was limited to data collected 
1n second-grade .classrooms (n ■ 90). 

Multiple data sources were available for composing the Indepen- 
dent variables, Including 'teacher questionnaire and Interview 
responses and brief (20 minutes) classroom observations during both 
reading and mathematics Instruction. 

Degree of Individualization 1n decisionmaking . Three variables 
were Included to operational 1ze the degree of Individualization 1n 
detffsYonmaking: sources used for placement, frequency of progress 
/monitoring, and frequency of remediation and/or corrective actions 
derived from progress-monitoring. 

Degree of Individualization by activity . 'During classroom ' 
^observations 1n both reading and mathematics, observers recorded the 
number of different activities occurring 1n the classroom. An^ 
activity was defined as a unique student assignment, often related to 
materials in use. For example, if some 'students were working on one 
workbook assignment while others were reading a text this would 
reflect two activities. However, if all students were working in the 
same workbook, but on three different assignments within the workbook, 
- then* this occurance would be recorded as three activities. 
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Degree of 1nd1v1dualizat1bn in teacher-student interactions . 
Teachers responded to questionnaire items asking what percentage of 
instructional time they typically spent in whole class, large group, 
small group, and individual instruction during reading and 
mathematics* " 

Number of aides and volunteers. Teachers indicated,, during 

. , — 

interviews, how many aides and/or volunteers assisted them during 
reading and during mathematics instruction; classroom observations 
also recorded the presence of aides and volunteers, 

Socio-ecpnomic status , SES was a school level index provided by 
the California State Department of Education, This three r point index 
was based on parent's occupation; three was the highest rating. 

Achievement measures ,-. Criterion-referenced tests of reading and 
mathematics were constructed specifically for the main study. 
Objectives were those agreed upon as central 1n the primary grade 
curriculum; their Importance was verified by teacher questionnaire 
responses. Because Individualized Instruction 1s supposed to permit 
all students to lea.m basic objectives, both level of achievement and 
classroom variation in achievement were included as variables of 
interest. 

Student attitudes . Items dealing with students 1 attitudes toward 
reading and toward mathematics were adapted from the School Sentiment 
Index (IOX, 1972). Three items were Included in the reading scale, 
and four Item? were included on the mathematics scale. 
Results * * 

Path, analysis" was used to* examine ttje significance of the i 

• * . 

hypothesized relations for both .reading and mathematics. To examine 
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whethej/ the patterns of relationships were the same for higher 
lower SES groups, Interaction terms were added^ the model. 
In reading, as predicted, ^odo-econom1c status was 

related to achievement, and whole class Instruction, for higher SES 

I i 

groups, was negatively related. However, teacher consulting with 
students and providing one measure of corrective action were 
negatively related to achievement, and whole class Instruction was 

associated with greater achievement for lower SES classrooms; these 

4 

latter findings are 1n direct contradiction to the concept of * 
Individualized Instruction. With respect to attk^ies toward riding 
consulting with students was a negative predictor^ p/esence,of\ 

more adults 1n lower SES classrooms was associated wltn more positive 

-\ 

student attitudes.- \ 
As expected, SES also was positively related to student - x 
performance 1n mathematics and v^hole class Instruction was negatively . 

i 

related to achievement. An unexpected finding Was that the number of 
adults was negatively related to achievement 1n lower SES classrooms." 
Grouping was found to contribute- both to mo^varlatlon 1n classroo^ 
achievement and to less positive attitudes toward mathematics. For 
lower SES classrooms, more activities and a teacher's' use of 
corrective action ,wer6 positively related to attitudes towards 
mathematics. 

These results showed some support for Individualized 
Instruction. The negative effect of whote class instruction on 
mathematics achievement, and on reading achlevment for students from 
higher socio-economic status backgrounds, supported one of the major 
premises of Individualized Instruction, I.e., providing only one 
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Instructional treatment 1s inappropriate to .the needs of many students 
within a class. The relationship between whole class instruction and 
w1th1n-class variation 1n reading achievement was similarly supportive 

r „ « 

of the premise. 

Hbwever, these»data suggest, as we might guess, that providing 
alternative materials and more~1nd1v1 dualized instructional settings 
does not splve the problem. of student learning. T&e relationship' 
between grou^1ng ? and 'within-class variation contradicts the theor- ^ • 
l «t1cal rhetoric, I.e., certain strategies associated with 1nd1v1du- 
an zed instruction may magnify differences among learners. The 
relationship between Individualized Instruction variables and the 
heading achievement^ for students from lower socio-economic backgrounds 
1s. particularly discouraging. /The results may Imply that, 1n current 
N /practice .xprov^dlng more Individualized strategies may not be appro- 
^1ate\for these students., a conclusion supported by research 1n 

teacher behavior and direct Instruction models (Rosenshine, 1977; 

v • . \ ' . 
nnett, 1976). 

"The relationship between process variables and student attitudes 

*" \ • \ 

was similarly contradictory. The results 1n mathematics suggest that 

while grouping practices associated with an Individualized approach 

may W detrimental to student attitudes for lower SES classrooms, 

other processes Which facilitate Instructional responsiveness and 

variety appear to enhance their attitudes. 

Despite the contradictions 1n the data and lack of relationship 

between most of the process variables and student outcomes, one 

conclusion js clear. One cannot assume that classrooms which appear 

more individualized are, 1n fact,, more facilltative environments for 
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students than are classrooms /which appear less Individualized. 
Processes underlying students 1 learning are metre complicated than 
surface appearances regarding 'teachers 1 use „ of assessment or t • 
provisions for Instructional alternatives* ' • 

This obvious conclusion has serious -implications for evaiuatlon 
policies at various levels. "For example, some SEAs and LEAs evaluate 
schools on the extent to which they provide Individualized Instruc- 
tion, based on brief classroom observations and Interviews (Herman & 

* * 

Hanelln, 1977). Certainly the validity of using such ratings to help 
assess school quality 1s suspect, a serious concern given the 
potential Impact of such practices on funding allocations. 

The mixed findings regarding the effects' of Individualized 
Instruction may be a function of the fact that classroom practice does 
not mirror the theory espoused by advocates of classroom Individuali- 
zation; that 1s, teachers do not truly Implement Individualized 
programs. Although teachers may look Hke they are Individualizing 
Instruction 1n terms of assessing students' progress and providing 
Instructional alternatives, etc., the results suggest that these 
actions are unrelated andrthat the link between diagnosis and 
prescription 1s missing-^ finding that again points to a need for 
• looking below the surface before passing evaluative judgment. 

What Insights does the example elaborated 1n this paper provide 
—beyond the not so astounding conclusion that molar variables often 
leave more questions than answers? I think 1t supports a few 
conclusions. 
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1. Evaluation of school programs, 'and particularly those • 
.programs' that focus on student achievement, bene/lt greatly 
from an Instructional research perspective* Evaluation^ 
often ask questions -that are too* simple; and* simple answers 
that Ignore schaol and 'classroom processes 'are likely to be 

>; 1'nvalld. Good dnfp'rmatlon requires a .knowledge of what goes' 
. ^on 1n classro^ns\and schools* ' 

Z\ Evaluation can provide good daja for Instructional research 
and 1t contributes to our knowledge base. The example 
reported he re^ perhaps does not repeal tojo much about 
Individualized Instruction, but the findings are consistent* 
with other findings 1n the field: for example, the results 
with regard to lower SES classrooms support much x>f the work 
of the direct Instruction advocates (see, for. example, 
StaTllngs et al, 1977; Soar, 1973; Rosenshlne,. 1977); that 
Individualized Instruction, 1n the example, tended to magnify 
differences between learners 1s consistent with some of the 
research from Wisconsin (personal correspondence, 1978). ^ 
Convergence of data from several studies certainly adds 
strength to the knowledge base. 

3. Evaluation studies which may necessarily have to look at more 
molar variables can. nonetheless support the need for more 
f1ne-gra1ned research. Similar findings across studies, 1n 
particular, provide a good rationale for why deeper 
understandings are necessary— and for the compelling need .for 
Instructional research. 

Is there a case for mutual needs? I think so. Evaluation 

certainly requires the research perspective, and we can benefit from 



the need as well as contribute to Informed, rather than. simplistic, 
public policy. 
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HITCHHIKING ON FAST-MOVING POLICY RESEARCH: A CRITIQUE 

Don Dorr-Bremme 

•i « 

Introduction 

This paper addresses two premises: (1) 1n a time of fiscal 
restraint," government funds are likely to be more available for policy 
studies and program evaluations while -grants for -basic research on 
teaching and learning become less available; /(2) researchers may try 
to "hitchhike" on government-sponsored policy studies— to use them as 
vehicles for doing research on Instruction. .These two premises raise, 
the question of whether research on teaching and learning can be built 
Into government-funded program evaluations and policy studies and, 1f 
so, -under what [circumstances? . — 

This paper addresses these questions' through a case study. It 
tells the story of how I tried to hitch a ride with, some research of 
my own on a policy'' study that happened, to 6e passing by 
elaborates the circumstances under which one can make g 
tlonal research' sense out of government policy, dollars. 

First I'll provide a description of the vehicle— a federally 
funded policy study— and the questions which drove 1t. Then 1*11 talk 
about the hitchhiker: me with my small piece of research that seemed 
to be, going in exactly the same direction as the vehicle. Finally, ; 
I'll .report how the ride seems to be going arid why. 
The Vehicle; A Federally Funded Study of Testing arid Test Use 
.. : The vehicle "passing by was a "piece of policy research: a 
national study to Inform federal, and especially state- and loc,al 
level policy, on achievement testing ♦ * 




, . Student^hlieyMenC^Stlng ,tn the nation's schools has become a 
vast enterprise* and tooth the amount.and variety of testing continue 
to. grow*. Across the. country, more ^?an '49 states have now mandated 
tesfc^fc.^ (Some states require 

the &st.s.{fb& S^ffptl^ii: andT- s^nduatrl'OQ s others, merely to check 

v.jrtUffiMjits 11 * < ^m|c- > Mfc#^jhBuQ^k-. needs at milestones 1n their school . 

'careers.) jThef testing, of student .achievement remains a primary way of 
meet : j|ng,thf'evaluat1on requirements, that federal and state programs 

• include-. School districts, have expanded their testing programs: many 
have developed or purchased assessment tools to monitor student's 
progress. along d1 : ?tr1ct-manda^d eontlnu'a of skills or' objectives. 
Teachers, meanwhile, deyelpp and administer their own tests as well as 
other, tests that come wjth cu#1cul urn materials they use. All 1n all, 
fftindreds of mfl-l tons. of dollars 1ri. public monies are expended annually 
oV*esting.# . Amidst thls-jtestlng^pom" variousttypes of tests, and 
testtfig .In igeheral iave byec^^ontroverslal./.tThe National 

Association and; the A^'&cffn Federation of. Te acher s,, for 
example, have, taken : of f1c1al A and ■somewhat opposing positions on 

-^test1|g.K' '-'' v { •:'•>;;•... k •..'"..'* 

^However, there has been very i*t£le research ^to Infd^in 'debate or 
decisions tt^'tMi^..j&CMjm testing 1s "going on, how much 
it costSi .and'what. specific. benefits derive from particular types of 

' % - It has' been estimated, for example, tha*-*r!976 standardized 
testing-in the elementary .grades alone cost well over a quarter of a 

" bill tofr d&l tars (tDC -News, 1£77). A study done by tyon (1978) found 
that budgets in; school districts' evaluation and testing units range 
between $2,006* and $4,000,000 annually.' These estimates, however, 

,-qmit substantial indirect costs: e.g., teachers' and administrators' 

**t1roe. spenOn preparation for testing, test administration, etc. 




tests and testing programs, under what circumstances, all remain 
largely unknown. The policy study on which I attempted to hitch a 
ride, then, since It would focus on test-policy Issues, had to address 
several broad questions: 

1. With what frequency and distribution are particular types <Jf 
tests given 1n the nation's schools? 

2. In what ways do particular types of tests and testing 
programs Impact upon schools arid those within them? 

" a) through their' very presence, required or recommended? 

b) through educators' utilization of their results? 

3. What factors Influence 

a) where and how much particular types of testing are done? 

' b) the ways that tests and their score? Impact on schools and 
those within them (students, teachers, etc.)? 

4. What are the costs--direct and Indirect dollar costs; 

* opportunity/ educational and psychological costs—of 
- different type's of < tests and testing programs? 

Of course, these research questions generate Information to 
address policy Issues such as: (1) What do we get and what do we trade 
off when we Invest our testing dollars 1n this, that, or some btjier 
test or assessment program? or (2) If we want to accomplish "X," 
what's the best Investment of our testing dollars? . 

These concerns and questions drove the policy stu<y, which took 
shaoe as a three-year effort. The first year would. entail planning 
the design of a national survey of teachers and principals (and some 
district officials). Exploratory fleidwork was Included, along with a 
literature review and reanalysls of some test-use data CSE had 
previously collected* The second year^ul d see Instrumentation and 

* . 4 * - 1 
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fielding of the survey 1n a national sample of districts^ over 100) 
and schools (Ideally two elementary and two high schools per 
district), for a total of .roughly 2100 respondents answering questions 
about testing and test use 1n the basic skills areas (reading/English 
and math). While data from this survey were being analyzed 1n the 
second year, planning for year three would begin, again Including, a 
good deal of't)n-s1te fleldwork. Finally, 1n the third year, 
ethnographic studies 1n three or four schools, as well as less 
Intensive fleldwork 1n other sites, would be carried .out to follow up 
on the survey and, especially, to get a close-up look at testing 
costs. 

This, then, was the vehicle— the policy study of testing and test 
use— and the, questions which drove' 1t. . One more point 1s worth noting 
about this before moving on. Although the words test and testing 
recur dbove, the study was equally concerned* from the outset with 
other, less formal means of assessment: teachers' observations andr 
dally Interactions and the Information they yield,, routine classwork 
and homework, etc. 

The Hitchhiker and His Study of Teachers' Thinking and Decision Making 

4 I turn now to the hitchhiker— myself— and the small piece of 
research I carried 1n my pack. First, 1t 1s Important to know that I 
have an Interest 1n what can be construed broadly as social 
cognition. More specifically, I'm concerned with understanding the 
everyday. knowledge- (c.f\, Sudnow, 1968), ,the "background 
understandings" (Ganwtkel, 1967)-, and practical reasoning (c.f., 
Cook-Gumperz, 1975) Wen are presumed to underlie the practical 
affairs of members of particular social groups. Put another way, 1n 



my work I attempt to describe the "system of standards for perceiving, 
believing., acting, and evaluating" (Goodenough, 1970, 1971) or the 
cognlflve "rules, maps, and plans" (Spradley,, 1972) evident In 
participants' routine actions and talk, and how these are functionally 
relevant {Erlckson,' 1978; Erlckson 4 Schultz, 1977) to the performance 
of particular educational events (e.g., lessons, morning circle time 
1n elementary classes, etc.)* 

These Interests are basically psychological 1n nature, but as the 
•language and citations may Indicate, Interests that I "pursue through 
the adjacent and sometimes complementary theories of cognitive 
anthropology, {Goodenough, 1964, 1971, 1975;' Tyler, 1969; Wallace, 
1970), ethnomethodol ogy (Clcourel, 1974; Garflnkel, 1967; Mehan & 
Wood,' 1975; Turner, 1974), and sodoHnguistlcs {Hymes, 19727-19741 
are merged 1n what Hugh Mehan has callkd "constitutive ethnography" 
and Fred Erlckson has termed "mlcroethnogr-apliy ." * 

It may by how be evident why the vehicle stopped to let me on. as 
a hitchhiker* I was not the designer of the, policy study, but I was 
clearly Interested 1n how things get done in schools and classrooms, 
1n how. people routinely think and act, and what, Influences how they 
think and act. Furthermore, I had the f1 el dwork training ajid 
experience that would be needed recurrently throughout the p/rlcy , 

study. '* ' /: ' . ». / 

• I, In. turn, was Interested 1n xHmblng aboard, for It seemed to > 
me that my research Interests were virtually congruent with the policy 
study's questions.. And there was plenty of fleldwork 1n the project 

% » ' * r 

* See for example, Bremme and Erlckson, 1977; Dorr-Bremme, 1981.(b). 



through which to pursue those Interests following the methodological 
canons of my field. More specifically, the policy study would seek'to 
determine what types of tests and other means of assessment .educators 
In schools use 'in making particular Instructional dedslops and how 
each "counts" as a given decision 1s made. That 1s essentially a, cog- 
nltlve' Issue, whlcli I could restate as: "What knowledge and processes 
of reasoning do teachers routinely employ as they make particular 
Instructional decisions?" Or again, "What system of standards for 
believing, perceiving, evaluating, and acting are routinely 1n use as. 
teachers make particular Instructional decisions?" I would summarize /\ 
those standards or cognitive VulesMn a flow chart such i a$/th6 
figure: . •« <j r.' 



Insert Figure 1 Abo.ut Here , A ^ 



• The flow chart here Is, of. course, more appropriate to. the meta-'. 
phor 1n this papfe> than to tfie study I wanted to dp.- But fn that ., 
study, times of day, kinds of people, driving conditions, etc., that * 
appeal 1n this chart would-be .replaced With kinds of te$ts arid other 
assessment, means, types of . students,. andUnstructlonal options that' 
Inform school decisions. Specifically,, ethnographic work would reveal 
how these elements figured ..In a particular instructional >; declslony- ' : 
where they fit 1n -the decision-making process, how much each .."counted" 
In the decision, ,etc. The study that lei to. these findings wbuld;.al.so 
tap teachers' "background understandings*— thelf opinions of testing .' 
1n general and of the ,worth of different, type's of tests ' and otherT 
assessment results. These Issues which woul<i provide, con.toxtf for the 
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findings summarized 1n my flow chart were also concerns of the policy 
study. 

>■ ' ' 

The data needs of my piece of research, then, would fie merely a 

• ■ ' <? * 

subset of the policy study. It would only be a matter of asking an 
extra question or two here and there In the .fleldwo.rk, looking a bit 
more closely at certain events we'd be looking at anyway,, and I'd 

>' 4. 

have done a reasonable piece of research on an aspect of teacher 

thinking and decision making that would be relevant to educational 

practice. This work wouldn't be as f1ne-gra1ned as usual constitutive 

or micro-ethnography, but 1t would use the same principles and 1t 

would be .supplemented with a great deal of Information from the larger 

policy study 1n "which 1t was embedded. ^ 

I didn't' decide, as I climbed aboard the policy vehicle, just • 

what Instructional decision I would focus *my research upon. I had In 

mind looking at the ways 1n whlph plassroom teachers decide a student 

needs extra -help. But I cotffd hold that "as a tentative choice pending 

S, 

the results of the first exploratory fieldwork. " 
— ■ »• 

In general, then, this hitchhiking would proceed as follows: 

(1) during- the first-year exploratory f1 el dwork, the goal would be to' 

get a general m,ap of the kinds, of ways teachers had of knowing about 

students' performance and progress: the*k1nds o£- decisions they made 

(as they saw them), and how the two seemed^ to relate. I would be 0 sure 

to Include questions on "special help" decisions 1n this work which 

would serve my substudy as background, and which we had to do anyway 

for the policy study. (All th1s)almost happened. Same 80 interviews 

were conducted with teachers, specialists, principals, and counselors, 

department chalrpeople, and others. Documents -were collected as well 

C ■ 



as copies of tests, and we tried to "look ethnographical 1y" 1n the • 
limited time we had 1n 'three schools 1n each of three districts.) 
(2) In the survey, I wanted to center Inquiry on the functional 
relevance of particular types of assessment results— tests and "» 
others— for particular "kinds of decisions. This , would be extremely 
Important for the policy study, 'as I saw 1t,, and it would (a) help me 
decide what sort of decision to focus my hitchhiking effort upon, as , 
well as (b) provide some broad data on that decision to contextual lie 
the fleldwork. (3) Then, 1n the fleldwork planning for the third 
year it and 1n the third year ethnographies themselves, I would begin In 

* 

earnest to do the extra little pieces of work which would give me my 
hitchhiking research m teacher thinking ,and decision making* « 

To this po-lnt, I have reviewed the policy study (the vehicle), 
how I hitchhiked on 1t and where I was sitting, and what I was 
carrying 1n my valise (the dedsl on-making study.) 
The R1de: Results of the Study and Some Caveats for' Hitchhikers 

In a short description of the ride I'd have to say that the t 
questions and concerns driving the policy study keep sneaking things 
from' the hitchhikers— time, money, and other resources— that they need 
to be methodologically robust. Both are still on the way to their 
destination. ^(Planning for the third year 1s concluding^ survey 
results have '.had preliminary analyses.) They may well. make 1t to 
where they are going, but the driver 1s getting awfully large, there 
" in' the front seat: room- for the hitchhiker seems to be shrinking. 

'less whimsically, data to set up the substudy as I Intended 
It— data which, ultimately will be a part of 1t If 1t does not fall off 
.the 'vehicle— Has begun to come 1n. Intuitively, the data seem solid 
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to me. But the documentation 1s weaker than I had hoped for,- less 
systematic than I feel good about. This has resulted, I think, from 
certain generic features of 'policy studies and circumstances 1n this 
particular study. I will describe these with examples, and underscore 
the lessons I think they teach. But first let me say something about 
the "findings," hypotheses really^at th'lV~ point, derived from the year 
one fleldwork, year two fleldwbrk to date, and some preliminary examl- 
nation of the survey questionnaire data* 

The picture that 1s emerging of how teachers routinely think 
about and handle student assessment 1s qblte consonant with the 
picture Eliot Frledsen paints of the "clinical mentality" In his soci- 
ology of applied knowledtge. Profession of Medicine , (1973) • In 
*■ 

unfortunately loaded language, Frledsen says, 

♦ 

The practitioner 1s a fairly crude pragmatlsty ,. prone 1n 
time to trust his own accumulation of personal first-hand 
experience In preference to abstract principles or "Oook ' 
knowledge", particularly 1n assessing and managing those 
aspects of his work that cannot be treated routinely. As 
Sharif and Levlnson noted in the case of psychiatrists 1n 
training, "The dangers of 1ntellectual1z1ng and "book learn- 
ing" are stressed. The highest >{alue 1s placed on emotional 
experience, on widening the range of "gut response" as a, 
means of understanding what- 1s going on 1n oneself and in 
the patient". This represents a certain subjectivism 1n his 
approach. > • 

Further on, Frledsen adds: "Thus, a rather thoroughgoing particular- 
Ism, a kind of ontologlcal and eplstemologlcal individualism Is 
characteristic of the clinician." Shed of Its pejorative language, 
Frledsen's description aptly "describes elements of structure 1n the 
teacher's thinking about assessment.* < . 



* The data which support the generalizations offered here may be 
found 1n Dorr-Br*emme, et. a1»» 1980; Dorr-Bremme, 1981.. 



1. Teachers' Thinking 1s Pragmatic and Experience Oriented . 

Our preliminary results show that the tests teachers give most 
often, devote most class time to, and rely upon roost heavily have 
three qualities: 

0 face validity— 1n the teacher's eyes, they match with what was 
actually taught 

0 Immediacy— they are Immediately available, may be given. 
d1scret1onar1ly, and the results are Immediately available 

0 consonance wi^h teacher's routine practical tasks— placement 
- test for placement; unit tests for unit grades; tests labeled 
diagnostic for diagnosis, etc?. 

Furthermore, clinical experience overrides, 1n many cases, test 

results. Almost Invariably, the teachers we spoke with said (without 

explicit el 1c1tat1 on from us) that they might use a placement test, 

* 

e.g., to group children for reading; but whether the placement was 

correct was determlned^on the basls'of.the teachers.' judgement of the 

child's work. Stillarly, according to teachers, some children are . 

"good test-takers;" others choke or may just not try; they may- be 

having a "bad day when the test Is g1ven> and so on. . 

2. Teachers* Reasoning About Test Results and Students'' Perfor- 
mance- in Particular ." r ~ r " ~\ ' 

The last point above Illustrates one form of particularism. 

Another emerged in the regularity of teachers repondlng "It • 

^depends.,." when^we asked them how a decision would be made. Family 

circumstances, classrpom social behavior, other teachers' remarks and 

opinions, oral performance, patterns Injroutine classwork, the appear- 

ance^f Interest or motivation, together with a wide variety of types 

Of test scores are available to teachers and figure 1n most decisions 

they make about their students. -Similar evidence, albeit organized 
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differently, also figured 1n teachers' assessments of their own per- 
formance—In their judgments of their effectiveness. It appeared— 
though 1t was difficult to tell certainly 1n Interviews— that this 
Information would be weighted differently, 1n making the same type of 
decision on different occasions or with different students. 

3. Teachers' Reasoning Processes Appear to be Rational when 
viewed within a cnnicai i-rameworic . 

Studies of teacher decision making offer conflicting results 

reagardlng how "rational" or valid teachers' clinical decisions are. 

Vlnsonhaler (1980, 1981), for instance, has demonstrated that the same 

reading specialist pften diagnoses an Individual student's "case" 

differently on two different .occasions separated in time; from 

specialist to specialist, there also seems to be little reliability 1n 

the diagnosis &f acase.-^On the other hand, similar, "policy 

■ t 

capturlng" studies using case simulations reported by Shavelson and ... 
colleagues (e.g., Borko,,Cone, Russo., 4 .Shavelson, 1979J Indicate that 
teachers can readily recognize' and usually tend to employ Information 
from more reliable sources. Other work (Pedul;la, Airaslan, * Madaus, 
1980) shows that teachers typically predict students' scores on 
. standardized tests quite accurately. Most research which has attended 
to the practical circumstances teachers confront as they make 
Instructional decisions tends to depict them as fairly reasonable. 
(For a comprehensive review, see Shavelson 4 Sterns, 1n press).. 

Work on teacher thinking and decision making conducted thus .far 
within the test use policy study tends to -support, and extend this view 
of the teacher. Exploratory findings, for example," suggest that , 
classroom practitioners rarely rely on one source of information 1n 
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making a given Instructional decision (Dorr-Brdmme, et al, 1980; Dorr- 
Bremme, 1981). Like the scientist, the teacher looks for replication 
of results generated through different types of measures ;tests of 
different types, class and homework assignments that embody different 
performance conditions, etc. This makes good sense 1n light of recent 
work In. human cognition (e.g., Griffin, Cole * Newman, 1n press), and 
language (Bloom & Gumperz 1972; Phillips 1973; Gumperz * Hernandez- 
Chavez, 1972) which shows that the demonstration of competence 1n 
performance varies with context. 

This 1s only a brief overview of findings to date, but 1t should 
Indicate that instructional research embedded within a pollcy^study 
can contribute to our collective understanding of teachers' thinking 
and .decision, making. As things stand, however, our findings are hot 
based on evidence as solid and as systematically obtained. as most 
researchers— even, ethnogs^phers, itfho are wrongly reputed to be less 
concerned with "harV data— would like, ^^ha^end^-lt 1s quite pbs- " 
sTble that our "findings" will remain provocative Impressions. They 
may not attain^thje status of research results 1n the stjjdy I'had hoped 
to do. Th1\s, as I noted earlier,' is largely because the policy study 
has consumed more resources I thought would be readily available for 
the hitchhiking research. - 

I do not think that this 1s a peculiarity of the policy study 1n 
which I am engaged. Policy studies are, I believe, a ravenous spe- 
cies. They are (to switch metaphors) generally subject to centrifugal 
forces'. In the remainder of this paper, I want to Indicate where 
these centrifugal forces come from and. then Illustrate with some exam- 
ples how they work to the disadvantage of a "hitchhiker." I will-also 



offer one or two caveats, drawn from my experience, for researchers 
who are considering hitchhiking on poUcy^studles with some research 
of their .own. 1 
The Nature of Policy Studies * 
► For a number of reasons, policy studies have a tendency to 'U«mt 

to be larger" than one originally anticipates they v/111 be. For 

■ 

example: • 
> J 

1. Funding 1s most often offered for research which 1s germane to 
national or statewide policy. The results, then, must be general - 
Izable to the nation or to a state or to other units which embody 
diverse program settings across a very large^umber of potentially 
program-relevant (and pol Icy-relevant) variables. Specifying a 
small number of* these, a priori, as sampling variables, 1s often 
difficult. This- has at least two Implications for research time 
and other resources: 

(a) There' 1s a tendency to be .Inclusive rather than exclusive: 
to sample along mone rather than fewer variables and thus to 

k expand a sample which was ratheY -large to begin with in order 
to obtain a sufficient n_ 1n each* of many sampling cells. This 
tendency has ramifications throughout the study. A larger 

••••* sample demands moire time for actually drawing the sample, for 
" , contacting the sites «to be surveyed and gaining their cooper- 
ation, for conducting the survey and managing the data, etc. 

• Wi There 1s rarely one single best way to draw a. natlona.1 or 
statewide sample for a given policy study. Alternative 
sampling plans offer the possibility of dlfferent-vbut equally 
policy-relevant— analyses. Examining these alternatives 
«vs requires, tlme^and review that may exceed, original projections, 
especial iy**hen project staff., representatives of the funding 
agency, reviewers, and consultants' d1 sagree on the merits of 
the different sampling plans and the analyses they facilitate. 

2. Policy research usually requires that data be, collected on a wide 
range of dependent variables. Previous studies have Identified a 
large 4 set of .generic factors that can Influence any program's 

" outcomes. (A partial 41st Includes leadership, participants 

feelings of program "ownership," the nature of the informal social 
structure of the implementing. Institution, participants "sense of 
efficacy," the number of other programs extant simultaneously at 
a site, participants 4 "angle of vision" dr perspective on the • 
program, and the frequency and quality of staff development and 
other support services.) Particular programs, of Course, are 
susceptible to the Influence of other variables 1n addition to. 
such generic ones. Thus, -the number of . domains relevant for 
i Inclusion 1n study Instruments 1s large. Furthermore,' a nation- 
wide or (to a lesser extent) a statewide sample, entails consider- 



able diversity 1n local conditions and practices, as well as 1n 
*•* local terminology for describing. conditions and practices. To 
'take this diversity Into account, questions 1n research Instru- 
ments must often be long and Com.pl ex, rather than simple and 
succinct. 

' 3. Policy research usually has clearly evident political Implica- 
tions. As a" consequence, policy- makers* and stakeholders 1n 
programs want to be sure their Interests and perspectives are 
' reflected 1n the research design and represented 1n the research 
Instruments. The funding agency may have one or multiple agendas • 
for- the study, some of which are politically motivated. Respond-. 
1ng to the concerns and wlches of various Interest groups 1s often 
unavoidable. In many cases, their Involvement 1n research plan- 
ning 1s critical to the success of the study. (For Instance, the 
support and/or endorsements of certain groups may facilitate local 
agencies' participation in the study, promote higher rates of 
return of survey questionnaires, etc. Support and endorsements 
may only come 1n exchange for a voice In research planning.) . 
Involving various Interested groups 1n planning and/or reviewing, 
the research design and Instruments consumes further time, energy, 
and research funds. Including questions that Interest groups, 
"suggest adds to the length of research Instruments. 

4. Policy research "usually must addYess multiple* audiences. Politi- 
cal considerations aside, these audiences do not always share the 
same concerns and questions. They may need and expect very d1f^ 
ferent kinds of ' Information. (The study of testing and test use, 
for example, Is expected to provide Information to policy makers 
1n federal, state, and local education agencies. Their Interests 

. • and Information needs are not Identical, however.) Balancing these 
competing Information needs Is another centrlfugaT force with- 
which policy research must contend. . Again, . there 1s a tendency to 
resolve the problem of multiple information needs by, making the 4 
research more Inclusive. . . 

5. When policy research 1s undertaken at a large scale (which it most 
often 1s when government-funded), a team effort 1s most often re- 
quired. Members of the research team may not agree on the best 
resolution of research issues** Compromising among researchers 
often results 1n expanding the scope of Inquiry. 

6. Today, requests for policy research frequently call for Inclusion 
of fieldwork of some klnd—ethnograpy, case studies, etc. On-site, 
work generates centrifugal forces: the closer one looks and the^ 
longer one looks on site, the more Issues seem to deserve Investi- 
gation. To accomodate these Issues, research Instruments and 
research time call but, for expansion. (\ . 

For a variety of reasons, then, there is a wndency' for policy 

research to spin out beyond Its projectedi>oundarjes. Even under the , 

best of circumstances, coping with these Centrifugal tendencies— - 
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Intelligently restraining the expansion of a policy study-*requ1res 
time,, staff energy, and Research dollars. And 1t seems that whep a 
substudy' 1s along for the, ride, 1t 1s that substudy that suffers first 
and most from the centrifugal forces. 
Some .Examples ' ' . . ' 

'My own experience provides several examples of how this happens. ( 
Earlier, I explained that my hitchhiking plan Included adding a ques- 
tlon .or \w to" the exploratory fleTdwork 1n the. test use study's first 
year. I had .also planned^ use that initial fieldwork as background 
for my teacher decision-making study.- To assure adequate time for 
this, I had/hppeti to conduct the fieldwork 1n states which had differ- 
ent testing 'requirements, {a -necessity for the pol icy study) ,, and .which 
were geographically close to California". Money saved in travel expen- 
ses-could then go 'toward making the exploratory work more systematic 
and rigorous. The funding agency*,. however., urged from the outset that 
the study be national. 1n scope. Responding to that suggestion was In 
Ch& best 1nteres> of good and continued relation between the agency m 
and our" research center. As a consequence, dollars were consumed 1n 
travel; time on site. was reduced.. to the minimum. 

*MoreoVer., long distance 'negotiations abut our site visits some- 
times resulted,- In unavoidable deviations from our research plan. We 
spoke on several occasions by phone *1th Jcey personnel 1n each d1s- 
trict we .'planned to visit* We al-soy exchanged several letters with 
them. During these contacts, we, stressed the Importance" of our speak- 
ing with each 1nterv1ewe*-f0r forty to forty-five minutes, and sugges- 
ted a number of measures, we were willing to take 1n order to arrange 
"for that. Each district acknowledged our request and. assured us that 
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they would respond to 1t. Neverthel essence we arrived on site, we 
found 1n several schools that Interviews were scheduled for shorter 
periods than we had requested/ Thus, we had to cut hack' our Interview, 
questions: thpse critical to the policy study remained; those addl- 1 
tlonal one or two questions most germane to /tine decision- making • 
study had to be cut. And with time on slteWready^at a minimum, 
there was no possibility of returning later to^gather Information 
lost. The exploratory fleldwork yielded a wealth of Information that 
proved extremely useful for designing the survey research which , 
followed. (That was Its primary function.) But with abbreviated 
Interviews 1n some schools and time on site focused on policy study 
Issues, the results were too asymetrlcally gathered to count, analyze, 
and Include as background 1n the substudy on teacher decisionmaking. 

Later on 1n the project, other centrifugal forces applied pres- 
sure on the substudy.^ In preparing for the national survey, two 
complete sampling plans were developed and discarded before arriving 
at a third and final one. This was not due to any Incompetence of the' 
plans' designers. (All of them were highly skilled with considerable 
experience 1n sampling for large-scale survey research. ) : The dlffiy^ 
culty was simply that each plan posited a slightly different set of 
variables (and survey analyses) as most Important. ' In successive 
reviews, the advocates of each plan and the consulting reviewers- 
argued effectively for 'the value of different sampling ^approaches. 
Each of these different points of view had. merits and drawbacks within 
the context o'f the study. Resolving these arid maximizing thV analyses 
the sample would" permit -required a good deal of time "and research 
dollars. All of this, of course, strengthened the policy study. But 



v 



- 49 - \^ 



1t delayed the start of the next phase of fieldwork, to be undertaken 

f * ... 

once sampling for the survey was underway. Once again, reduced time 
meant restricting the focus of fleldwork to Issues most germane to the 
policy study, and passing over the extra Uttle bit of work which 
would havfe fleshed 'out the research on teacher dedsl on-making. / 

In thb survey ItseJf, I had hoped to focus .the Inquiry; on the 
functional relevance, of different types of assessment results— test 
scores and other student products— on teachers' decisions. A number 
of pressures came to bear, however, which minimized the. attention 
which could be\g1ven to this domain of Inquiry. First*, a large number 
of domains had to be covered on the questionnaire, and jqueitlons on 
each domain grew longer 1n order to take Into account the diversity of 
practices across. the. nation. Project officers fii the funding agency 
emphasized that certain areas were of critical Importance, given agen- 
cy Information needs. Representatives off teachers 1 organizations, 
commercial test publishers, and others Involved (for political rea- 
.sons) 1n the review process called for Inclusion of certain types of 
questions. Project staff members argued effectively for different 
emphases. To avoid an extensive burden on questionnaire respondents, 
difficult choices were necessary. In the end, collective/ thinking and 
conflicting demands assured comprehensiveness 1n the survey. Ques- 
t1<?ns on how teadhers'used particular types of assessnjent results were 
Included, but only as one of several domains of Inquiry Important to* 
th^e policy study. " , , v 

These were only^ome^of the ways 1ij^1ch the centrifugal forces 
Inherent 1n policy research compressed 1^Pec1s1ort-making stucjy I 
planned to conduc^THtajt these examples shoula be sufficient to 111 us- 
trate the general process. 

* » 
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I should add that this was not my first experience with policy . 
research: I was aware that each phase and task of the project would 
tend to expand 1n scope and complexity, and had anticipated, that 
tendency as I planned my hitchhiking. But there seemed to - be so much 
overlap between the issues of the policy study and the Issues of my 
dedsl on-making study that 1t seemed I would*be able to address 'both 
even though the former were likely to grow as work continued. 
CaveatSj ^ . 

V 

TJie observatlpns I have made here about the nature of policy 

\_ * 

research are unlikely to be new to those experienced 1n such work; 
they routinely experience the centrifugal forces I have described. 
But as grants for research become scarce, .scholars new to the policy 

4 

research road will be stepping onto 1t 1n greater numbers In search of 

vehicles on which they can hitchhike with their own Instructional ly, 

relevant studies." For then, I offqr the«fqllow1ng wprds of advice. 

It Is probably best not to hitchhike wfth strangers. That 1s: 

- "tv- Your research on instruction 1s likely to lose weight" to the 
■ extent tnaf Its questions are not exactly congruent both -with 
_Jthe research questions of the' policy study and with the 
"actual items 1n research Instruments. ' . 

2. Get a ride for your study on the research 'methods that are 
absolutely central to the policy study, no matter how exten- 
sive some other methods may appear to be 1n the research . 
' .design. ' , . 

> • • 

The substud(y research discussed 1n th1s % paper was accomplished 

effectively largely because 1t followed these caveats. Large-scale 

policy resear<fh>^hen 1t 1s done reasonably, provides very. little room 

for naive* bl"tchh1kers. . , • 
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